Method and apparatus for image acquisition including image sensor

ABSTRACT

An image acquisition apparatus includes a display layer including hole regions through which external light is received and pixel regions arranged between the hole regions, an image sensor disposed under the display layer and configured to generate a raw image by sensing the external light received through the hole regions, and a processor configured to perform image processing on the raw image based on blur information based on an arrangement of the hole regions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is continuation of application Ser. No. 17/230,000filed on Apr. 14, 2021, which claims the benefit under 35 USC § 119(a)of Korean Patent Application No. 10-2020-0110128 filed on Aug. 31, 2020,and Korean Patent Application No. 10-2021-0031386 filed on Mar. 10,2021, in the Korean Intellectual Property Office, the entire disclosureof which is incorporated herein by reference for all purposes.

BACKGROUND 1. Field

The following description relates to a method and apparatus for imageacquisition including an image sensor.

2. Description of Related Art

A camera, a device configured to capture an image, is widely equipped invarious electronic devices. A camera has become an essential part of amobile device, such as a smartphone, and becomes more advanced andsmaller in size over time. In general, a smartphone may include a frontcamera and a rear camera. The front camera may disposed at the upper endof the smartphone for capturing a user's selfie.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

In one general aspect, an image acquisition apparatus includes a displaylayer including hole regions through which external light is receivedand pixel regions arranged between the hole regions, an image sensordisposed under the display layer and configured to generate a raw imageby sensing the external light received through the hole regions, and aprocessor configured to perform image processing on the raw image basedon blur information based on an arrangement of the hole regions.

Each of the hole regions may be arranged between a subset of the pixelregions in the display layer, and be arranged such that a minimumdistance between a boundary of a hole region and a boundary of a pixelregion is within ⅕ of a diameter of the hole region.

Each of the hole regions may be circular and the hole regions may be thesame in shape and size.

Each of the hole regions may be greater in size than each of the pixelregions, and the hole regions may be separated from each other.

The processor may generate a preprocessed image by performing imagepreprocessing on the raw image, and generate an enhanced image byperforming image restoration on the preprocessed image.

The processor may be further configured to generate the preprocessedimage by performing either one or both of image preprocessing configuredto apply a filter to the raw image and image preprocessing configured toperform image deconvolution on the raw image.

The processor may be further configured to obtain the enhanced imageusing a neural network-based image restoration model configured to usethe preprocessed image as an input.

The blur information may be determined based on any one or anycombination of any two or more of a size, a shape, a depth, or anarrangement of the hole regions.

The blur information may include information associated with a blur ofthe raw image that is determined by at least one of the shape, size,depth, or interval of the hole regions.

A spacing between centers of neighboring ones of the hole regions may besubstantially equal to a diameter of any one of the hole regions.

A ratio of a spacing between centers of neighboring ones of the holeregions and a diameter of any one of the hole regions may besubstantially equal to 1.

In another general aspect, an image acquisition apparatus includes adisplay layer including pixel regions and hole regions of a circularform configured to allow external light to be received therethrough, andan image sensor disposed under the display layer and configured togenerate an image by sensing the external light. Each of the holeregions is arranged between a subset of the pixel regions in the displaylayer. The hole regions and the pixel regions are alternately arrangedin the display layer, and the hole regions are each of a same shape andsize.

Neighboring hole regions may be separated from each other with a wiringregion for wiring therebetween, and four hole regions may be arranged tosurround a single pixel region.

In another general aspect, an image sensor includes a processorconfigured to generate a raw image based on external light receivedthrough hole regions comprised in a display layer, and perform imageprocessing on the raw image based on blur information based on anarrangement of the hole regions.

Each of the hole regions may be arranged between a subset of pixelregions in the display layer, and be arranged such that a minimumdistance between a boundary of a hole region and a boundary of a pixelregion is within ⅕ of a diameter of the hole region. Each of the holeregions may be greater in size than each of the pixel regions, and thehole regions may be separated from each other.

In another general aspect, an electronic apparatus includes a displaypanel including a display layer having hole regions through whichexternal light is received and pixel regions for outputting a displayimage, an image sensor disposed under the display panel and configuredto generate a raw image by sensing the external light, a processorconfigured to generate an enhanced image by performing image processingon the raw image based on blur information based on an arrangement ofthe hole regions, and a storage configured to store either one or bothof the raw image and the enhanced image.

Each of the hole regions may be arranged between a subset of the pixelregions in the display layer. Each of the hole regions may be greater insize than each of the pixel regions, and the hole regions are separatedfrom each other.

In another general aspect, a method of operating an image acquisitionapparatus, includes obtaining a raw image by sensing external lightreceived through hole regions arranged in a display layer using an imagesensor disposed under the display layer, and performing image processingon the raw image based on blur information based on an arrangement ofthe hole regions.

The performing of the image processing may include generating apreprocessed image by performing image preprocessing on the raw image,and generating an enhanced image by performing image restoration on thepreprocessed image.

The generating of the preprocessed image may include generating thepreprocessed image by performing either one or both of imagepreprocessing configured to apply a filter to the raw image and imagepreprocessing configured to perform image deconvolution on the rawimage.

The performing of the image processing may include obtaining theenhanced image using a neural network-based image restoration modelusing the preprocessed image as an input.

A non-transitory computer-readable storage medium may store instructionsthat, when executed by one or more processors, cause the one or moreprocessors to perform the method above.

In another general aspect, an image acquisition apparatus include adisplay layer comprising hole regions and pixel regions arranged in apredetermined pattern, an image sensor disposed under the display layerand configured to generate a raw image by capturing light through thehole regions, and a processor configured to process the raw image basedon blur information of the hole regions.

The predetermined pattern may include a subset of the hole regionssurrounding each of the pixel regions.

The predetermined pattern may include the hole regions interleaved withthe pixel regions.

Each of the hole regions may be circular, the hole regions may be of asame shape and size, and each of the hole regions may be greater in sizethan each of the pixel regions.

Each of the hole regions may be a micro-hole region.

The blur information may be determined based on any one or anycombination of any two or more of a size, a shape, a depth, anarrangement of the hole regions, or a point spread function (PSF) basedon the arrangement of the hole regions.

The display layer may be included in a display panel of an electronicapparatus.

A spacing between centers of neighboring ones of the hole regions may besubstantially equal to a diameter of any one of the hole regions.

A ratio of a spacing between centers of neighboring ones of the holeregions and a diameter of any one of the hole regions may besubstantially equal to 1.

The processor may be further configured to generate a preprocessed imageby performing either one or both of image preprocessing configured toapply a filter to the raw image and image preprocessing configured toperform image deconvolution on the raw image.

The raw image may be a demosaiced red green blue (RGB) image.

The processor may be further configured to generate an enhanced image byperforming image restoration on the preprocessed image.

The enhanced image may be generated using a neural network-based imagerestoration model configured to use the preprocessed image as an input.

The image processing apparatus may further include an image signalprocessor (ISP) configured to perform any one or any combination of anytwo or more of noise reduction, white detect correction, RGB shading,RGB interpolation, color correction, and image format conversion on theenhanced image.

Other features and aspects will be apparent from the following detaileddescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of an electronic apparatus having an imageacquisition apparatus and an example of an enlarged capturing region ona display screen.

FIG. 2 illustrates an example of a structure of an image acquisitionapparatus.

FIGS. 3 and 4 illustrate an arrangement of hole regions arranged in adisplay layer.

FIG. 5 illustrates a point spread function (PSF) based on an arrangementstructure of hole regions.

FIG. 6 illustrates an example of a change in a PSF based on anarrangement structure of hole regions.

FIGS. 7A through 7C illustrate image restoration.

FIG. 8 illustrates an example of a flowchart of an image processingmethod.

FIG. 9 illustrates an example of training an image restoration model.

FIG. 10 illustrates an example of an image acquisition apparatus.

FIG. 11 illustrates an example of an image sensor.

FIG. 12 illustrates an example of an electronic apparatus.

Throughout the drawings and the detailed description, the same referencenumerals refer to the same elements. The drawings may not be to scale,and the relative size, proportions, and depiction of elements in thedrawings may be exaggerated for clarity, illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader ingaining a comprehensive understanding of the methods, apparatuses,and/or systems described herein. However, various changes,modifications, and equivalents of the methods, apparatuses, and/orsystems described herein will be apparent after an understanding of thedisclosure of this application. For example, the sequences of operationsdescribed herein are merely examples, and are not limited to those setforth herein, but may be changed as will be apparent after anunderstanding of the disclosure of this application, with the exceptionof operations necessarily occurring in a certain order. Also,descriptions of features that are known after understanding of thedisclosure of this application may be omitted for increased clarity andconciseness.

The features described herein may be embodied in different forms, andare not to be construed as being limited to the examples describedherein. Rather, the examples described herein have been provided merelyto illustrate some of the many possible ways of implementing themethods, apparatuses, and/or systems described herein that will beapparent after an understanding of the disclosure of this application.

The terminology used herein is for the purpose of describing particularexamples only, and is not to be used to limit the disclosure. As usedherein, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. As used herein, the term “and/or” includes any one and anycombination of any two or more of the associated listed items. As usedherein, the terms “include,” “comprise,” and “have” specify the presenceof stated features, numbers, operations, elements, components, and/orcombinations thereof, but do not preclude the presence or addition ofone or more other features, numbers, operations, elements, components,and/or combinations thereof.

In addition, terms such as first, second, A, B, (a), (b), and the likemay be used herein to describe components. Each of these terminologiesis not used to define an essence, order, or sequence of a correspondingcomponent but used merely to distinguish the corresponding componentfrom other component(s).

Throughout the specification, when an element, such as a layer, region,or substrate, is described as being “on,” “connected to,” or “coupledto” another element, it may be directly “on,” “connected to,” or“coupled to” the other element, or there may be one or more otherelements intervening therebetween. In contrast, when an element isdescribed as being “directly on,” “directly connected to,” or “directlycoupled to” another element, there can be no other elements interveningtherebetween. Likewise, expressions, for example, “between” and“immediately between” and “adjacent to” and “immediately adjacent to”may also be construed as described in the foregoing.

Unless otherwise defined, all terms, including technical and scientificterms, used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this disclosure pertainsconsistent with and after an understanding of the present disclosure.Terms, such as those defined in commonly used dictionaries, are to beinterpreted as having a meaning that is consistent with their meaning inthe context of the relevant art and the present disclosure, and are notto be interpreted in an idealized or overly formal sense unlessexpressly so defined herein.

Also, in the description of example embodiments, detailed description ofstructures or functions that are thereby known after an understanding ofthe disclosure of the present application will be omitted when it isdeemed that such description will cause ambiguous interpretation of theexample embodiments.

Hereinafter, examples will be described in detail with reference to theaccompanying drawings, and like reference numerals in the drawings referto like elements throughout.

FIG. 1 illustrates an example of an electronic apparatus having an imageacquisition apparatus and an example of an enlarged capturing region ona display screen.

In FIG. 1 , an image acquisition apparatus configured to obtain imagedata by capturing an image may operate by being embedded in anelectronic apparatus 110. The image acquisition apparatus may operate bybeing embedded in the electronic apparatus 110 including a display 120.As the electronic apparatus 110, any type of electronic apparatusincluding a display may be used unrestrictedly. The use of the term‘may’ herein with respect to an example or embodiment, e.g., as to whatan example or embodiment may include, implement, or achieve means thatat least one example or embodiment exists with such a feature,implementation, or achievement, while also noting that all examples andembodiments are not limited thereto and alternate examples orembodiments may also exist.

A camera of the image acquisition apparatus that receives external lightmay not be externally exposed but disposed inside the electronicapparatus 110. The camera of the image acquisition apparatus may bedisposed under the display 120 of the electronic apparatus 110. An imageacquisition apparatus with a camera disposed as described above is alsoreferred to as an under-display camera (UDC). Since the camera isdisposed inside the electronic apparatus 110, it is possible to includea region where the camera is disposed as a display region. Thus, it ispossible to implement a quadrangular display without a need to implementa notch-type display or dispose an independent camera region inside adisplay region.

For example, in an example where an image sensor is disposed inside oneregion 130 of the display 120, a display layer 140 corresponding to theregion 130 may include a plurality of pixel regions 150 and a pluralityof hole regions 160, each being of a circular form, as illustrated inFIG. 1 . However, the form of the hole regions 160 is not limited to theillustrated circular, and the hole regions 160 may be embodied invarious forms, such as, for example, an elliptic form and a quadrangularform. A hole region described herein is also referred to as a micro-holeregion. The arrangement pattern of the pixel regions 150 and the holeregions 160 may be duplicated in the display layer 140 corresponding tothe region 130. As illustrated, each of the hole regions 160 may bearranged between the pixel regions 150 and arranged near the pixelregions 150. In an example, a hole region 160 may be arranged such thata minimum distance between a boundary of the hole region 160 and aboundary of a pixel region 150 is within ⅕ of a diameter of the holeregion 160. However, the detailed numerical value, ⅕, is provided merelyas an example, and thus the scope of examples is not limited thereto.

The image acquisition apparatus may obtain image data based on externallight received through the hole regions 160 of the display layer 140. Animage may be output through or displayed on the pixel regions 150,including other pixel regions included in another region of the display120. The display layer 140, a component included in a display panel, maybe a layer in which the pixel regions 150 are arranged in a redeterminedpattern. As illustrated, the hole regions 160 through which externallight pass into the electronic apparatus 110 may be present only in theregion 130 in which the image acquisition apparatus is disposed.Although the region 130 and the display layer 140 in which the holeregions 160 are arranged are illustrated in a circular form, the region130 and the display layer 140 may also be provided in various forms.

FIG. 2 illustrates an example of a structure of an image acquisitionapparatus.

FIG. 2 depicts an example of a cross-section of the region 130 of theelectronic apparatus 110 of FIG. 1 . In FIG. 2 , an image acquisitionapparatus includes a display layer 210 and an image sensor 220. Thedisplay layer 210 includes a plurality of pixel regions 230 configuredto output a color and a plurality of hole regions 240 of a circular formthat allows external light 250 to be received therethrough. The pixelregions 230 and the hole regions 240 may be alternately arranged on thedisplay layer 210 in the region 130 of the electronic apparatus 110 ofFIG. 1 . The remaining region of the display layer 210, from which thehole regions 240 are excluded, may all be configured to inhibit lightfrom passing therethrough. Thus, external light 250 may only reach theimage sensor 220 after passing through the hole regions 240.

On the display layer 210, a protective layer 260 of a transparent ortranslucent material is disposed to protect the display layer 210. Theprotective layer 260 may be formed with tempered glass or reinforcedplastic, for example. In addition, the display layer 210 may includeother components to implement a display panel in addition to the pixelregions 230. A display including such pixel regions 230 may be embodiedby a display type, such as, for example, a liquid crystal display (LCD)and an organic light-emitting diode (OLED).

The image sensor 220 may be disposed under the display layer 210 andconfigured to generate a raw image by sensing the external light 250received through the hole regions 240. According to examples, the imagesensor 220 may be designed to be ultra-small and provided as a pluralityof image sensors. For example, the raw image, which is generated by theimage sensor 220 through light passing through the hole regions 240, mayinclude a demosaiced red, green, blue (RGB) image. The external light250 reaching the image sensor 220 may be a portion of light incident onthe display layer 210 and passing through the hole regions 250. Thus,the raw image obtained by the image sensor 220 may have an image qualitylevel that is less than a desired image quality level. For example, theraw image may have a relatively low level of brightness and a relativelysignificant amount of noise due to occlusion by the pixel regions 230.In addition, the hole regions 240 may act as slits, and there may thusbe artifacts in the raw image due to a diffraction effect. For example,the raw image may have a blur or a flare.

Due to such image quality degrading factors, image processing in astructure such as a UDC may be desired to enhance the raw image obtainedby the image sensor 220. The image acquisition apparatus may furtherinclude a processor configured to perform such image processing.According to examples, the image processing may be performed in theimage sensor 220. The image processing may include restoring the rawimage obtained by the image sensor 220 to have a similar image qualityto that of a typical image captured by a camera that does not includehole regions 240. The image acquisition apparatus may perform such imagerestoration by processing the image based on an arrangement (e.g., ashape, size, depth, interval, etc. of the hole regions 240) of the holeregions 240, and may thus provide a high-definition clear image even inan environment using the UDC.

The arrangement of the hole regions 240 included in the display layer210 may be determined based on the shape, size, depth, interval, and thelike of each of the hole regions 240. It may be an arrangement optimizedfor the image processing or optimized to reduce the image qualitydegrading factors in the raw image. Such an optimized arrangement may bedetermined based on a blur characteristic exhibited by the arrangementof the hole regions 240. Examples of the arrangement of the hole regions240 will be described hereinafter with reference to FIGS. 3 and 4 .

FIGS. 3 and 4 illustrate a portion of a display layer in which aplurality of hole regions is arranged but there may be differentarrangements of the hole regions in other examples.

In FIG. 3 , a plurality of pixel regions 312, 314, 316, and 318 arearranged throughout a display layer, and a plurality of hole regions320, 322, 324, 326, and 328 are arranged only in a portion of thedisplay layer. In the display layer, the hole regions 320, 322, 324,326, and 328 and the pixel regions 312, 314, 316, and 318 may bealternately arranged.

For example, each of the hole regions 320, 322, 324, 326, and 328 mayhave a circular contour, and have the same shape and size. For example,the hole region 320 of a single circular form may be arranged among thepixel regions 312, 314, 316, and 318. Each hole region may be arrangednear to four pixel regions along with four other hole regions in thedisplay layer. For example, the hole region 320 may be arranged near tothe other hole regions 322, 324, 326, and 328, and to the pixel regions312, 314, 316, and 318. Neighboring hole regions may be separated fromone another by a wiring region for wiring therebetween, and four holeregions may be arranged to surround a single pixel region. The holeregions 320, 322, 324, 326, and 328 may be designed to have a maximumsize in a region in the display layer from which the pixel regions 312,314, 316, and 318 and the wiring region are excluded. The hole regions320, 322, 324, 326, and 328 may be arranged near the pixel regions 312,314, 316, and 318. For example, the hole region 320 may be arranged suchthat a minimum distance between a boundary of the hole region 320 and aboundary of each of the pixel regions 312, 314, 316, and 318 that isnear the hole region 320 is within ⅕ of a diameter of the hole region160.

FIG. 4 illustrates another example of an arrangement of a plurality ofhole regions in a display layer. In FIG. 4 , similar to the examplearrangement illustrated in FIG. 3 , a plurality of hole regions 420,422, 424, 426, and 428 and a plurality of pixel regions 412, 414, 416,and 418 may be alternately arranged in a display layer. The hole regions420, 422, 424, 426, and 428 each provided in a circular form may havethe same shape and size. Each of the hole regions 420, 422, 424, 426,and 428 may be arranged near four neighboring hole regions and fourpixel regions. The hole regions 420, 422, 424, 426, and 428 may beidentified from each other by a wiring region therebetween, and may bedesigned to have a maximum size in a region from which the pixel regions412, 414, 416, and 418 and the wiring region are excluded.

As described above with respect to FIGS. 3 and 4 and the examplearrangements of hole regions, it is possible to allow more externallight to be received through hole regions of a maximum size. It is thuspossible to obtain a high-brightness raw image based on theconfiguration of the hole regions. In addition, through an arrangementstructure in which hole regions of the same size and shape are arrangednear one another, it is possible to reduce the degradation of an imagequality that may occur due to diffraction. An optimal arrangementstructure for hole regions may be obtained by a process of determining abasic form of a hole region of a curved shape in a state in which pixelregions satisfying a given display requirement are arranged in a displaylayer, a process of arranging a hole region between pixel regionsarranged farthest among neighboring pixel regions and pixel regionsarranged near to a current pixel region, a process of increasing a sizeof arranged hole regions until the hole regions are to be within apreset distance from a pixel region and a wiring region for wiring, anda process of blocking all micro-hole regions that may be generated by adisplay structure other than corresponding hole regions.

An image acquisition apparatus may generate a higher-definition image byperforming image preprocessing on a raw image by considering an opticalcharacteristic based on an arrangement structure of hole regionsdetermined as described above, and by performing image restoration.Further details regarding such image processing will be described below.

FIG. 5 illustrates a point spread function (PSF) based on an arrangementstructure of hole regions.

FIG. 5 illustrated are one region 505 having a diameter 510 in a displaylayer, a plurality of hole regions including hole regions 520 and 525,and a plurality of pixel regions including a pixel region 515. A blur ofa raw image to be sensed by an image sensor through hole regions may besimulated as a PSF 540 based on the hole regions' arrangement structure.

The PSF 540 may be a mathematical or numerical representation indicatinghow a single pixel region or point to be included in the raw imagespreads. Through the PSF 540, it is possible to estimate blurinformation to be indicated in the raw image. A shape of the PSF 540 mayvary according to a size of the hole regions, a shape of the holeregions, a depth of the hole regions, an interval between the holeregions, and/or an arrangement form of the hole regions on atwo-dimensional (2D) plane. For example, the size of the hole regionsmay determine an envelope form of the PSF 540 and a distance 550 to amain lobe. An interval 530 between the neighboring hole regions 520 and525 may determine the position and intensity of a first side lobe. Inaddition, a ratio of the interval between the hole regions 520 and 525to the size of each of the hole regions 520 and 525 may determine theintensity of the first side-lobe.

For example, as the size of the hole regions 520 and 525 increases, thedistance 550 between a center 542 and the main lobe may increase, andthe intensity of the first side lobe may decrease. As the interval 530between the hole regions 520 and 525 decreases, a distance 545 betweenthe center 542 and the first side lobe may increase, and the intensityof the first side lobe may decrease. Further details regarding therelationship between an arrangement structure of hole regions and a PSFwill be described below.

FIG. 6 illustrates an example of a change in a PSF based on anarrangement structure of hole regions. In the example of FIG. 6 , it isassumed that hole regions in cases 610 and 630 have the same diameter w,corresponding to a size of each of the hole regions in both cases 610and 630, but the interval (or spacing) s1 between the hole regions incase 610 is greater than the interval s2 between the hole regions incase 630. It is also assumed that other conditions are the same. In theexample of FIG. 6 , also illustrated are PSFs 620 and 640 that arederived by simulating the arrangements of the hole regions in cases 610and 630, respectively.

Comparing the PSFs 620 and 640, as the interval between the hole regionsdecreases, the distance from a center (e.g., 622, 642) to a first sidelobe increases, and the intensity of the first side lobe decreases. Thefirst side lobe indicates an element corresponding to a blur or anafterimage in a raw image, and thus the intensity of the first side lobeneeds to decrease.

The envelope of a PSF may be determined by the size of a hole region.For example, in PSFs 620 and 640, since each of the hole regions havethe same size, the envelope of PSFs 620 and 640 may be determined basedon diameter w of any hole region of the hole regions. In PSFs 620 and640, as the spacing s1 or s2 approaches the diameter w, or the ratio ofs1/w or s2/w approaches 1, the first side lobe will approach null.

Referring back to FIG. 5 , based on a relationship between thearrangement of the hole regions and the form of the PSF 540, it may bedesirable that the size of each of the hole regions increases, and theinterval 530 between the neighboring hole regions 520 and 525 decreases.In addition, it may be desirable that the hole regions are alternatelyarranged with the pixel regions and arranged near each of the pixelregions, and each has a form of a single circle. Also, when determiningthe arrangement of the hole regions, a margin for wiring and the likebetween the neighboring hole regions 520 and 525 may be considered.

From the PSF 540 based on the hole regions' arrangement structure, blurinformation that may be indicated in a raw image may be estimated, andan image processing method based on the arrangement structure may beemployed. For example, in an example where the interval 530 between thehole regions 520 and 525 is relatively large, a strong double image mayoccur. In such an example, the image acquisition apparatus may remove adouble image or a blur from the raw image using information associatedwith the PSF 540 and then perform image restoration. For anotherexample, in an example where the interval 530 between the hole regions520 and 525 is relatively small, strong noise and a widened blur contourmay occur. In such an example, the image acquisition apparatus mayperform deconvolution, or image restoration using a neural network.

FIGS. 7A through 7C illustrate image restoration.

In FIG. 7A, under a display layer 710 in which a plurality of pixelregions 720 and a plurality of hole regions 715 are arranged, an imagesensor 730 may generate raw data by sensing external light receivedthrough the hole regions 715. In operation 735, blur information isextracted from an arrangement of the hole regions 715 arranged in thedisplay layer 710. The blur information may vary based on a shape, size,depth, and/or interval. The blur information may be obtained from a PSFdetermined through a simulation or an equation.

In operation 740, image preprocessing is performed on the raw data basedon the blur information. The image preprocessing may include an imageprocessing method to reduce artifacts such as a double image or a blurincluded in the raw data. The image preprocessing may include, forexample, filtering that applies a filter to a raw image, deconvolutionprocessing using a known kernel, or a neural network model-based methodto obtain an image with a reduced double image or blur.

In operation 745, image restoration may be performed on a preprocessedimage obtained through the image preprocessing. The image restorationmay allow a result from the image restoration to be more similar to animage obtained by an image sensor in an environment in which there is noportion occluded by the pixel regions 720. For the image restoration, atrained neural network-based image restoration model may be used. Theimage restoration model may use the preprocessed image as an input andoutput an enhanced image obtained through the image restoration.

To obtain a desirable result from the image preprocessing and the imagerestoration, the size of each of the hole regions 715 arranged in thedisplay layer 710 may need to be as large as possible, and a distancebetween the hole regions 715 may need to be as short as possible. Theenhanced image obtained through the image restoration may be transferredto an image signal processor (ISP) 750. The ISP 750 may process theenhanced image in a desirable way. For example, the ISP 750 may performimage processing, for example, noise reduction, white detect correction,RGB shading, RGB interpolation, color correction, image formatconversion, or the like.

In FIG. 7B, the image preprocessing and the image restoration that aredescribed above may be performed after image processing is performed byan ISP 755 on a raw image. The raw image is obtained by the image sensor730, and the obtained raw image is transferred to the ISP 755. The ISP755 performs the image processing, for example, noise reduction, on theraw image. Subsequently, image preprocessing corresponding to operation740 is performed on an image obtained through the image processingperformed by the ISP 755 in operation 760, and then the imagerestoration is performed in operation 765.

In FIG. 7C, the image preprocessing and the image restoration that aredescribed above may be performed on a raw image in operations 775 and780 between image processing by a first ISP 770 and image processing bya second ISP 785. The raw image obtained by the image sensor 730 istransferred to the first ISP 770, and the first ISP 770 performs theimage processing, for example, noise reduction, on the raw image.Subsequently, the image preprocessing is performed based on blurinformation in operation 775, and the image restoration is performed inoperation 780. A resulting image obtained through the image restorationis transferred to the second ISP 785, and the ISP 780 performs, on theresulting image, the image processing, for example, noise reduction,white defect correction, RGB shading, RGD interpolation, colorcorrection, image format conversion, or the like.

FIG. 8 illustrates an example of a flowchart of an image processingmethod.

In FIG. 8 , in operation 810, an image acquisition apparatus obtains araw image using an image sensor. The image sensor may be disposed undera display and configured to generate the raw image by sensing externallight received or captured through a plurality of hole regions of adisplay layer.

In operations 820 and 830, the image acquisition apparatus generates anenhanced image by performing image processing on the raw image based onblur information that is based on an arrangement of the hole regions. Astructure of the arrangement of the hole regions may be determined bythe shape, size, depth, and/or interval of the hole regions. Inoperation 820, the image acquisition apparatus generates a preprocessedimage by performing image preprocessing on the raw image based on theblur information. For example, the image acquisition apparatus maygenerate the preprocessed image by performing at least one of imagepreprocessing that applies a filter to the raw image or image processingthat performs image deconvolution on the raw image.

In operation 830, the image acquisition apparatus generates the enhancedimage by performing image restoration on the preprocessed image. Forexample, the image acquisition apparatus may obtain the enhanced imageusing a neural network-based image restoration model using thepreprocessed image as an input. The image restoration model may be amodel trained to output, as output data, the enhanced image obtainedthrough the image restoration performed on the preprocessed image whenthe preprocessed image is input as input data. Further details regardingthe training of the image restoration model will be described below.

FIG. 9 illustrates an example of training an image restoration model.

In FIG. 9 , a neural network-based image restoration model may betrained based on a typical image and a preprocessed image. The typicalimage may be generated as an image sensor 920 senses external lightreceived through a display layer 910 in which a main aperture 915, notmicro-hole regions, is present. The typical image generated as describedin the foregoing may be stored in a typical image database (DB) 930.

A raw image may be obtained as an image sensor 960 senses external lightreceived through a plurality of hole regions 945 of a display layer 940in which the hole regions 945 and a plurality of pixel regions 950 arealternately arranged. An arrangement of the hole regions 945 and thepixel regions 950 in the display layer 940 may be the same as describedabove with respect to the accompanying drawings. The image sensor 960may be the same as the image sensor 920. In the example of FIG. 9 , itis assumed that the image sensors 920 and 960 have no other differencesin obtaining an image, except a difference between the display layers910 and 940. In operation 970, image preprocessing is performed on theraw image obtained by the image sensor 960. The preprocessed imagegenerated through such image preprocessing may be stored in apreprocessed image DB 980. The image preprocessing may be performedbased on blur information obtained from an optical characteristic thatis based on the arrangement of the hole regions 945, and includeremoving a double image from the raw image using the blur information.The blur information may include information associated with a PSFdetermined based on the arrangement of the hole regions 945.

In operation 990, training of the image restoration model is performedbased on the typical image and the preprocessed image. The preprocessedimage may be input to the image restoration model for image restorationto be performed, and a result image obtained through the imagerestoration may be compared to that of a typical image obtained in atypical environment. Parameters of a neural network included in theimage restoration model may be updated such that a difference betweenthe result image and the typical image defined by a loss function isreduced based on the loss function. Through such a process describedabove, the training may be performed such that the result image outputfrom the image restoration model is more similar to the typical image.

The process may be performed repeatedly on a plurality of sets oftraining data, and the image restoration model may be updated to havemore desirable results. According to examples, synthetic data obtainedby artificially adding noise or a blur to the typical image may be usedto train the image restoration model. For example, the synthetic datamay be input to the image restoration model, and parameters of the imagerestoration model may be updated such that a difference between a resultimage output from the image restoration model and the typical imagebefore synthesis is reduced.

FIG. 10 illustrates an example of an image acquisition apparatus.

In FIG. 10 , an image acquisition apparatus 1000 includes a displaylayer 1010, an image sensor 1020, and a processor 1030. The displaylayer 1010 may include a plurality of hole regions through whichexternal light is received and a plurality of pixel regions. The pixelsare arranged between the hole regions. On the display layer 1010, thehole regions and the pixel regions may be alternately arranged. Each ofthe hole regions may have a greater size than each of the pixel regions,and be provided in the form of a single circle. The hole regions mayhave the same shape and size, and be identified from each other by awiring region therebetween.

The image sensor 1020 may be disposed under the display layer 1010, andgenerate a raw image by sensing the external light received through thehole regions, and transmit the generated raw image to the processor1030. The image sensor 1020 may include a camera configured to receiveexternal light and generate image data.

The processor 1030 may control an overall operation of the imageacquisition apparatus 1000, and execute instructions to perform one ormore operations described above with reference to FIGS. 1 through 9 .For example, the processor 1030 may generate an enhanced image byperforming image processing the raw image based on blur information thatis based on an arrangement of the hole regions. The blur information mayinclude information association with a blur of the raw image that isdetermined by a shape, size, depth, and/or interval of the hole regionspresent in the display layer 1010. The blur information may include, forexample, information associated with a PSF based on a structure of thearrangement of the hole regions. The processor 1030 may generate apreprocessed image by performing image preprocessing on the raw imageobtained by the image sensor 1020, and generate the enhanced image byperforming image restoration on the preprocessed image. For the imagepreprocessing, the processor 1030 may perform image preprocessing thatapplies a filter to the raw image and/or image preprocessing thatperforms image deconvolution on the raw image to generate thepreprocessed image. The processor 1030 may obtain the enhanced imageusing a neural network-based image restoration model that uses thepreprocessed image as an input.

FIG. 11 illustrates an example of an image sensor.

In an example, an image sensor 1110 may not only obtain a raw image butalso generate an enhanced image by performing image processing on theobtained raw image. Thus, the enhanced image may be output from theimage sensor 1110.

In FIG. 11 , the image sensor 1110 includes a camera 1120 and aprocessor 1130. The camera 1120 may generate a raw image by sensingexternal light received through a plurality of hole regions included ina display layer. In the display layer, the hole regions and a pluralityof pixel regions may be alternately arranged. Each of the hole regionsmay have a greater size than each of the pixel regions, and the holeregions may be identified from each by a wiring region therebetween. Forexample, each of the hole regions may be arranged near to four pixelregions along with four other hole regions.

The processor 1130 may generate an enhanced image by performing imageprocessing on the raw image based on blur information that is based onan arrangement of the hole regions. The processor 1130 may perform imagepreprocessing, for example, filtering, on the raw image obtained by thecamera 1120, and perform image restoration on a preprocessed imageobtained through the image preprocessing based on a neural network-basedimage restoration model. As a result of the image restoration, theenhanced image may be generated.

FIG. 12 illustrates an example of an electronic apparatus.

An electronic apparatus 1200, which is an apparatus including a display,may be, for example, a smartphone, a tablet computer, a wearable device,a netbook, a laptop, or the like. In FIG. 12 , the electronic apparatus1200 includes a processor 1210, a memory 1220, an image sensor 1230, astorage device 1240, an input device 1250, an output device 1260, and acommunication device 1270. Such components of the electronic apparatus1200 may communicate with one another through a communication bus 1280.The electronic apparatus 1200 may perform all the functions of the imageacquisition apparatus 1000 described above with reference to FIG. 10 .

The processor 1210 may control an overall operation of the electronicapparatus 1200 and execute functions and instructions to be performed inthe electronic apparatus 1200. The processor 1210 may perform one ormore, or all, of operations or methods described above with reference toFIGS. 1 through 11 . The processor 1210 may generate an enhanced imageby performing image processing on a raw image obtained through the imagesensor 1230. For example, the processor 1210 may generate the enhancedimage by performing image preprocessing and image restoration on the rawimage based on blur information that is based on an arrangement of holeregions in a display layer.

The memory 1220 may store information necessary for the processor 1210to perform its processing operation. For example, the memory 1220 maystore instructions to be executed by the processor 1210, and storerelated information during the execution of software or an applicationin the electronic apparatus 1200. The memory 1220 may include, forexample, a random-access memory (RAM), a dynamic RAM (DRAM), a staticRAM (SRAM), or other types of nonvolatile memory that are well-known inthe related technical field.

The image sensor 1230 may be disposed under a display panel, includingthe display layer, and configured to generate the raw image by sensingexternal light received through the hole regions arranged in the displaylayer. The image sensor 1230 may include a camera to receive theexternal light, and may also perform image processing on the raw imageaccording to examples.

The storage device 1240 may include a computer-readable storage mediumor device, and store the raw image and the enhanced image. The storagedevice 1240 may include, for example, a storage, a magnetic hard disk,an optical disc, a flash memory, an electrically erasable programmableread-only memory (EEPROM), or the like.

The input device 1250 may receive, from a user, an input, for example, atactile input, a video input, an audio input, or a touch input. Theinput device 1250 may include, for example, a keyboard, a mouse, atouchscreen, a microphone, a retinal scanner, and other devices that maydetect the input from the user and transmit the detected input to theelectronic apparatus 1200.

The output device 1260 may provide an output of the electronic apparatus1200 to a user through a visual, auditory, or tactile channel. Theoutput device 1260 may include, for example, s display panel for aliquid crystal display (LCD) and a light-emitting diode (LED)/organiclight-emitting diode (OLED) display, a touchscreen, a speaker, avibration generator, and other devices that may provide the output tothe user. For example, in an example where the output device 1260 is adisplay panel, the display panel may include a display layer in which aplurality of hole regions through which external light is received and aplurality of pixel regions for outputting a display image are arranged.

The communication device 1270 may communicate with an external devicethrough a wired or wireless network. The communication device 1270 mayreceive and transmit data or information from and to the externaldevice.

As a non-exhaustive example only, an electronic apparatus 110 asdescribed herein may be a mobile device, such as a cellular phone, asmart phone, a wearable smart device (such as a ring, a watch, a pair ofglasses, a bracelet, an ankle bracelet, a belt, a necklace, an earring,a headband, a helmet, or a device embedded in clothing), a portablepersonal computer (PC) (such as a laptop, a notebook, a subnotebook, anetbook, or an ultra-mobile PC (UMPC), a tablet PC (tablet), a phablet,a personal digital assistant (PDA), a digital camera, a portable gameconsole, an MP3 player, a portable/personal multimedia player (PMP), ahandheld e-book, a global positioning system (GPS) navigation device, ora sensor, or a stationary device, such as a desktop PC, ahigh-definition television (HDTV), a DVD player, a Blu-ray player, aset-top box, or a home appliance, or any other mobile or stationarydevice configured to perform wireless or network communication. In oneexample, a wearable device is a device that is designed to be mountabledirectly on the body of the user, such as a pair of glasses or abracelet. In another example, a wearable device is any device that ismounted on the body of the user using an attaching device, such as asmart phone or a tablet attached to the arm of a user using an armband,or hung around the neck of the user using a lanyard.

The image acquisition apparatus, the electronic apparatus, and otherapparatuses, devices, units, modules, and components described hereinwith respect to FIGS. 1-12 are implemented by hardware components.Examples of hardware components that may be used to perform theoperations described in this application where appropriate includecontrollers, sensors, generators, drivers, memories, comparators,arithmetic logic units, adders, subtractors, multipliers, dividers,integrators, and any other electronic components configured to performthe operations described in this application. In other examples, one ormore of the hardware components that perform the operations described inthis application are implemented by computing hardware, for example, byone or more processors or computers. A processor or computer may beimplemented by one or more processing elements, such as an array oflogic gates, a controller and an arithmetic logic unit, a digital signalprocessor, a microcomputer, a programmable logic controller, afield-programmable gate array, a programmable logic array, amicroprocessor, or any other device or combination of devices that isconfigured to respond to and execute instructions in a defined manner toachieve a desired result. In one example, a processor or computerincludes, or is connected to, one or more memories storing instructionsor software that are executed by the processor or computer. Hardwarecomponents implemented by a processor or computer may executeinstructions or software, such as an operating system (OS) and one ormore software applications that run on the OS, to perform the operationsdescribed in this application. The hardware components may also access,manipulate, process, create, and store data in response to execution ofthe instructions or software. For simplicity, the singular term“processor” or “computer” may be used in the description of the examplesdescribed in this application, but in other examples multiple processorsor computers may be used, or a processor or computer may includemultiple processing elements, or multiple types of processing elements,or both. For example, a single hardware component or two or morehardware components may be implemented by a single processor, or two ormore processors, or a processor and a controller. One or more hardwarecomponents may be implemented by one or more processors, or a processorand a controller, and one or more other hardware components may beimplemented by one or more other processors, or another processor andanother controller. One or more processors, or a processor and acontroller, may implement a single hardware component, or two or morehardware components. A hardware component may have any one or more ofdifferent processing configurations, examples of which include a singleprocessor, independent processors, parallel processors,single-instruction single-data (SISD) multiprocessing,single-instruction multiple-data (SIMD) multiprocessing,multiple-instruction single-data (MISD) multiprocessing, andmultiple-instruction multiple-data (MIMD) multiprocessing.

The methods illustrated in FIGS. 1-12 that perform the operationsdescribed in this application are performed by computing hardware, forexample, by one or more processors or computers, implemented asdescribed above executing instructions or software to perform theoperations described in this application that are performed by themethods. For example, a single operation or two or more operations maybe performed by a single processor, or two or more processors, or aprocessor and a controller. One or more operations may be performed byone or more processors, or a processor and a controller, and one or moreother operations may be performed by one or more other processors, oranother processor and another controller. One or more processors, or aprocessor and a controller, may perform a single operation, or two ormore operations.

Instructions or software to control computing hardware, for example, oneor more processors or computers, to implement the hardware componentsand perform the methods as described above may be written as computerprograms, code segments, instructions or any combination thereof, forindividually or collectively instructing or configuring the one or moreprocessors or computers to operate as a machine or special-purposecomputer to perform the operations that are performed by the hardwarecomponents and the methods as described above. In one example, theinstructions or software include machine code that is directly executedby the one or more processors or computers, such as machine codeproduced by a compiler. In another example, the instructions or softwareincludes higher-level code that is executed by the one or moreprocessors or computer using an interpreter. The instructions orsoftware may be written using any programming language based on theblock diagrams and the flow charts illustrated in the drawings and thecorresponding descriptions in the specification, which disclosealgorithms for performing the operations that are performed by thehardware components and the methods as described above.

The instructions or software to control computing hardware, for example,one or more processors or computers, to implement the hardwarecomponents and perform the methods as described above, and anyassociated data, data files, and data structures, may be recorded,stored, or fixed in or on one or more non-transitory computer-readablestorage media. Examples of a non-transitory computer-readable storagemedium include read-only memory (ROM), random-access programmable readonly memory (PROM), electrically erasable programmable read-only memory(EEPROM), random-access memory (RAM), dynamic random access memory(DRAM), static random access memory (SRAM), flash memory, non-volatilememory, CD-ROMs, CD−Rs, CD+Rs, CD−RWs, CD+RWs, DVD-ROMs, DVD−Rs, DVD+Rs,DVD−RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-rayor optical disk storage, hard disk drive (HDD), solid state drive (SSD),flash memory, a card type memory such as multimedia card micro or a card(for example, secure digital (SD) or extreme digital (XD)), magnetictapes, floppy disks, magneto-optical data storage devices, optical datastorage devices, hard disks, solid-state disks, and any other devicethat is configured to store the instructions or software and anyassociated data, data files, and data structures in a non-transitorymanner and provide the instructions or software and any associated data,data files, and data structures to one or more processors or computersso that the one or more processors or computers can execute theinstructions. In one example, the instructions or software and anyassociated data, data files, and data structures are distributed overnetwork-coupled computer systems so that the instructions and softwareand any associated data, data files, and data structures are stored,accessed, and executed in a distributed fashion by the one or moreprocessors or computers.

While this disclosure includes specific examples, it will be apparentafter an understanding of the disclosure of this application thatvarious changes in form and details may be made in these exampleswithout departing from the spirit and scope of the claims and theirequivalents. The examples described herein are to be considered in adescriptive sense only, and not for purposes of limitation. Descriptionsof features or aspects in each example are to be considered as beingapplicable to similar features or aspects in other examples. Suitableresults may be achieved if the described techniques are performed in adifferent order, and/or if components in a described system,architecture, device, or circuit are combined in a different manner,and/or replaced or supplemented by other components or theirequivalents.

Therefore, the scope of the disclosure is defined not by the detaileddescription, but by the claims and their equivalents, and all variationswithin the scope of the claims and their equivalents are to be construedas being included in the disclosure.

What is claimed is:
 1. An electronic apparatus comprising: a displaycomprising hole regions and pixel regions arranged in a predeterminedpattern; a camera disposed under the display and configured to obtain araw image by sensing external light received through the hole regions;and a processor configured to generate an enhanced image by performingimage restoration on the raw image, wherein boundaries of the holeregions are respectively separated from boundaries of the pixel regions.2. The electronic apparatus of claim 1, wherein the processor is furtherconfigured to perform an image processing on the raw image to reduceartifacts by the hole regions.
 3. The electronic apparatus of claim 1,wherein the processor is further configured to: perform imagerestoration on the raw image based on a blur information, wherein theblur information is based on an arrangement of the hole regions.
 4. Theelectronic apparatus of claim 3, wherein the blur information isdetermined based on any one or any combination of any two or more of ashape, size, depth, and interval of the hole regions, or a point spreadfunction (PSF) based on the arrangement of the hole regions.
 5. Theelectronic apparatus of claim 1, wherein a spacing between centers ofneighboring ones of the hole regions is substantially equal to adiameter of any one of the hole regions.
 6. The electronic apparatus ofclaim 1, wherein each of the hole regions is arranged between the pixelregions.
 7. The electronic apparatus of claim 1, wherein each of thehole regions is circular and the hole regions are of a same shape andsize.
 8. The electronic apparatus of claim 1, wherein each of the holeregions is greater in size than each of the pixel regions.
 9. Theelectronic apparatus of claim 1, wherein the predetermined patterncomprises a subset of the hole regions surrounding each of the pixelregions.
 10. The electronic apparatus of claim 1, wherein thepredetermined pattern comprises the hole regions interleaved with thepixel regions.
 11. The electronic apparatus of claim 1, wherein aspacing between centers of neighboring ones of the hole regions issubstantially equal to a diameter of any one of the hole regions. 12.The electronic apparatus of claim 1, wherein the processor is furtherconfigured to: generate a preprocessed image by performing either one orboth of image preprocessing configured to apply a filter to the rawimage and image preprocessing configured to perform image deconvolutionon the raw image.
 13. The electronic apparatus of claim 12, wherein theprocessor is further configured to: obtain the enhanced image using aneural network-based image restoration model configured to use thepreprocessed image as an input.
 14. The electronic apparatus of claim 1,wherein the processor is further configured to: perform any one or anycombination of any two or more of noise reduction, white detectcorrection, red green blue (RGB) shading, RGB interpolation, colorcorrection, and image format conversion on the enhanced image.
 15. Amethod of operating an electronic apparatus, comprising: obtaining a rawimage by sensing external light received through hole regions arrangedin a display using a camera disposed under the display; and generatingan enhanced image by performing image restoration on the raw image,wherein the hole regions and pixel regions of the display are arrangedin a predetermined pattern, and wherein boundaries of the hole regionsare respectively separated from boundaries of the pixel regions.
 16. Themethod of claim 15, wherein the generating of the enhanced imagecomprises: performing an image processing on the raw image to reduceartifacts by the hole regions.
 17. The method of claim 15, wherein thegenerating of the enhanced image comprises: performing image restorationon the raw image based on a blur information, wherein the blurinformation is based on an arrangement of the hole regions.
 18. Themethod of claim 15, wherein the generating of the enhanced imagecomprises: generating a preprocessed image by performing either one orboth of image preprocessing configured to apply a filter to the rawimage and image preprocessing configured to perform image deconvolutionon the raw image.
 19. The method of claim 18, wherein the generating ofthe enhanced image comprises: obtaining the enhanced image using aneural network-based image restoration model using the preprocessedimage as an input.
 20. A non-transitory computer-readable storage mediumstoring instructions that, when executed by one or more processors,cause the one or more processors to perform the method of claim 15.